Real-Time Video-Based Modeling and Rendering of 3D Scenes

virtual reality, developing techniques for synthesizing arbitrary views has become an important technical issue. Given an object’s structural model (such as a polygon or volume model), it’s relatively easy to synthesize arbitrary views. Generating a structural model of an object, however, isn’t necessarily easy. For this reason, research has been progressing on a technique called image-based modeling and rendering (IBMR) that avoids this problem. To date, researchers have performed studies on various IBMR techniques. (See the “Related Work” sidebar for more specific information.) Our work targets 3D scenes in motion. In this article, we propose a method for view-dependent layered representation of 3D dynamic scenes. Using densely arranged cameras, we’ve developed a system that can perform processing in real time from image pickup to interactive display, using video sequences instead of static images, at 10 frames per second (frames/sec). In our system, images on layers are view dependent, and we update both the shape and image of each layer in real time. This lets us use the dynamic layers as the coarse structure of the dynamic 3D scenes, which improves the quality of the synthesized images. In this sense, our prototype system may be one of the first full real-time IBMR systems. Our experimental results show that this method is useful for interactive 3D rendering of real scenes.